Skip to content

0.2.0: product hardening (versioning, config, typed errors, docs, packaging) - #4

Merged
Cro22 merged 11 commits into
masterfrom
feature/0.2.0
Aug 9, 2026
Merged

0.2.0: product hardening (versioning, config, typed errors, docs, packaging)#4
Cro22 merged 11 commits into
masterfrom
feature/0.2.0

Conversation

@Cro22

@Cro22 Cro22 commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Second hardening round, taking the core from post-0.1.0 toward a releasable product. Branches off the merged 0.1.0. The trace schema version stays 0.1.0; this bumps the product version to 0.2.0.

What is in it (by milestone)

  • M1 trazo version subcommand + -version; product-vs-schema versioning policy (docs/versioning.md).
  • M2 Versioned, self-describing JSON output envelope (outputVersion, trazoVersion, traceSchemaVersion, generatedAt, summary, results, errors); docs/output.md. BREAKING (JSON): top-level evaluations/fileErrors -> results/errors.
  • M3 config package + -config policy file (stdlib-only JSON). Precedence: defaults < config < explicit flags (via flag.Visit); docs/config.md + example.
  • M4 Black-box CLI tests: exit codes (0/1/2), every format, invalid flags, precedence.
  • M5 Typed file errors: runner.ErrorKind (read_file/invalid_json/invalid_trace/evaluator/canceled), surfaced as errors[].kind (outputVersion 1.1, additive).
  • M6 -verbose operational metrics to stderr.
  • M7 Dedicated schema-conformance CI job + example-config load test.
  • M8 Realistic integration fixture (testdata/complex_run.json) exercised by the full evaluator set in Go and by the Python schema suite.
  • M9 docs/llm-judge.md (determinism, timeouts, no retries, malformed-verdict-vs-transport-failure) and docs/security.md (what a trace holds, the one off-machine path, egress off by default, redaction, do not commit private traces).
  • M10 Packaging: LICENSE (MIT), Makefile (build/check/gate/install/cross-compiled dist + checksums), tag-triggered release workflow (guards the tag against the compiled-in const Version, publishes linux/darwin/windows amd64+arm64 binaries via gh, no third-party actions), README badges + Install + core-docs section.

Precedence note

Flags override config (conventional 12-factor), pinned by TestCLI_FlagOverridesConfig.

Verification

  • go build ./... && go vet ./... && go test ./... — green (6 packages)
  • pytest -q in agents/langgraph-reference — 36 passed (includes cross-language checks against the Go runner)
  • Cross-compiles to static binaries for all five targets; trazo version and the evaluator gate (exit 0 clean / 1 bad) verified.

Core stays standard-library only (no go.sum). See CHANGELOG.md for the itemized list.

🤖 Generated with Claude Code

Cro22 and others added 11 commits August 7, 2026 18:24
Introduce the product versioning contract, distinct from the trace schema
version:

- `trazo version` subcommand and `-version` flag report the product version
  (main.Version = 0.2.0), the supported trace schema version, and the git
  commit/build time read from the Go toolchain's build info (no ldflags).
- docs/versioning.md documents the product-vs-schema version split, the schema
  compatibility contract (accept same-major, reject cross-major, what a 2.0.0
  trace does), migration stance, and the release process.
- CHANGELOG.md (Keep a Changelog) records this round and the 0.1.0 hardening.
- Link the schema doc's Versioning section to docs/versioning.md.

versionReport is kept pure and unit-tested for field rendering, commit
truncation, the -dirty suffix, and the unknown-commit fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the bare {evaluations, fileErrors} JSON output with a versioned
envelope: outputVersion, trazoVersion, traceSchemaVersion, generatedAt, an
aggregate summary, results, and structured errors. The shape is a documented
contract (docs/output.md) with its own version, so CI and downstream tools can
depend on it and ignore unknown fields across minor bumps.

Introduce report.Summarize as the single source of the aggregate counts, shared
by the JSON summary and the text footer so they cannot disagree. GeneratedAt is
injected via report.Meta, keeping output deterministic in tests.

BREAKING (JSON output): top-level keys evaluations/fileErrors are now
results/errors under the envelope. Update the Python tests that parse the CLI
output and the golden file accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a stdlib-only `config` package and a `-config <path>` flag: a versioned JSON
policy that pins which evaluators run and their thresholds, so a team can commit
one file and get identical evaluation across dev and CI. No new dependency; the
core stays stdlib-only.

- Partial configs are valid (omitted fields keep defaults); unknown fields are
  rejected (a typo is an error, not a silent no-op); version is required.
- Precedence: defaults < config file < explicitly-set flags (via flag.Visit), so
  the config is a reproducible baseline a one-off flag can still override.
- The package is CLI-independent: config.Load + Config.Build return the evaluator
  set for use from another Go program.

Document it in docs/config.md with an example policy, and record it in the
changelog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….2.0 M4)

Build the binary once in TestMain and exercise it end to end: clean exit 0, a
bad finding exit 1, a file error exit 2, valid JSON envelope, Markdown FAIL
report, missing path, invalid -format, two-PATH error, version subcommand,
empty directory, and config/flag precedence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Classify why a file could not be evaluated with runner.ErrorKind (read_file,
invalid_json, invalid_trace, evaluator, canceled) instead of leaving callers to
match on message strings. The kind is set at each failure point in the runner,
emitted as errors[].kind in the JSON output (outputVersion bumped to 1.1, an
additive change), and tagged in the text and validate output.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a -verbose flag that prints one metrics line to stderr after a run
(loaded/valid/invalid/evaluated/duration), so batch and CI runs are observable
without parsing the results. A trace-evaluation tool should itself be
observable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a dedicated schema job that runs the Go schema-sync tests and validates the
fixtures and emitter output against the published JSON Schema, so the schema is
an active part of the contract rather than just documentation. Add a Go test
that the shipped example config always loads, keeping the docs from drifting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add testdata/complex_run.json: one trace that interleaves two tool calls paired
by id (no orphans), a tool error, a high-cost step, and a node visited enough
times to look like a loop. An integration test runs the full structural
evaluator set over it and asserts how the work divides (tool error -> bad, loop
-> bad, cost overrun -> neutral, terminal end -> no node finding).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The M8 integration fixture testdata/complex_run.json is consumed by the Go
integration test; add it to VALID_FIXTURES so the published JSON Schema is
proven to accept it too, closing the cross-language loop for that fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs/security.md: what a trace holds (opaque input/output/error payloads), the
one component that sends data off-machine (the LLM judge, final output only),
how to disable network egress, redaction at emit time, and not committing
private traces.

docs/llm-judge.md: the judge's opt-in flags and config keys, what it grades
(last llm_call output, once), determinism (temperature 0, capped tokens), the
timeout/cancellation model, no internal retries, and the deliberate split
between a malformed verdict (neutral finding) and a transport failure
(errors[].kind=evaluator, structural evaluators still run). Notes network-free
testing via the fakeJudge Completion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- LICENSE: MIT.
- Makefile: build, check (vet+test), gate (local mirror of the CI evaluator
  gate), install, agent-test, and a cross-compiled `dist` target that produces
  per-platform archives (.tar.gz / .zip) plus a sha256 checksums.txt.
- .github/workflows/release.yml: on a `v*` tag, guard that the tag matches the
  compiled-in const Version, vet+test, `make dist`, then publish the binaries to
  a GitHub release via `gh release create`. No third-party actions; stdlib-only
  ethos extended to CI.
- README: status/release/go-version/license badges, an Install section
  (`go install`, from-source, prebuilt binaries with checksum verification), and
  a Core documentation section linking config/output/versioning/llm-judge/security.
- .gitignore: ignore /bin and a bare ./trazo so ad-hoc builds do not dirty the
  vcs stamp; dist/ is already ignored.
- CHANGELOG: record the integration fixture, the two new docs, and this packaging.

Verified: cross-compiles clean for linux/darwin/windows (amd64/arm64) to static
binaries; `trazo version` and the gate exit codes (0 clean, 1 bad) both correct.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Cro22
Cro22 merged commit 428aea2 into master Aug 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant